RESTRICTION TO SYMMETRIC SUBGROUPS OF UNITARY 
REPRESENTATIONS OF RANK ONE SEMISIMPLE LIE GROUPS 

B. SPEH AND G. ZHANG 

Abstract. We consider the spherical complementary series of rank one Lie groups 

f»^ ■ H n = SOo(n, 1; F) for F = M, C, H. We prove that there exist finitely many discrete 

£SJ | components in its restriction under the subgroup H n _\ = SOo(n — 1, 1;F). This is 

5^ ■ proved by imbedding the complementary series into analytic continuation of holomorphic 

O-t, discrete series of G n = SU(n,l), SU(n,l) x SU(n,l) and SU(2n,2) and by the 

•^ ■ branching of holomorphic representations under the corresponding subgroup G n -\. 

o 
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1. Introduction 



This paper is a continuation of []22| by Speh-Venkataramana and ll26l by Zhang. Con- 
sider the complementary series representations (7r M , H) of the Lie group H = SO (n, 1; F) 
for F = E, C, HI and its restriction to the subgroup Hi = SO (n — 1, 1; F). It is proved 
there that for certain parameters ji the restricted representation has a direct summand 
which is isomorphic to a complementary series representation of the smaller group H\\ 
similar results are proved also for the exceptional Lie group H = SO (2, 1;0) := 
00 ! ^4(-20), and #1 = Spin(8, 1). 

In |[22| the complementary series (n^,H) of H = SOo(n,l; M.) are realized in its 
non-compact picture, namely as appropriate spaces of distributions on the space R 71 ^ 1 
and it is proved that, roughly speaking, up to the Fourier transform, the trivial extension 
of distributions on the subspace M n ~ 2 to distributions W 1 " 1 defines an Hi -intertwining 



operator and thus determines for proper values of /1 a discrete summand of (7r M , H). 

The general rank one case is treated in [|26l using the compact picture realizing the 
complementary series as certain Sobolev type spaces on the sphere in F n , the discrete 
components being obtained by restricting the functions to the sub-sphere in F n_1 . This 
requires some detailed estimates for the branching of representations of the maximal 
compact subgroup K C H on the space of spherical harmonics on F n under the compact 
subgroup L C Hi. 

In the present paper we determine additional discrete summands of the restriction of 
(iTfj,, H) to H\. These are also isomorphic to complementary series representations of Hi 
and we determine explicitly the parameters of these complementary series representations 
of H\. See theorem 5.1 for a precise statement. 
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2 B. SPEH AND G. ZHANG 

In the proof we shall use a rather different idea than in the previous papers. We consider 
H respectively Hi as subgroups of a larger groups G, respectively G\ so that we have the 
following commutative diagram of subgroup inclusions 

G G x 



H H x 

Here we shall consider the Hermitian groups G = SU(n, 1), SU(n, 1) x SU(n, 1), 
SU(2n,2). 

We start with an scalar holomorphic representation (U u , G) of G realized on a Hilbert 
space of holomorphic functions on the unit ball G/K, and consider its branching along 
the diagram. Denote (£/£, G\) the scalar holomorphic representations of G\. The restric- 
tion of ([/„, G) to G\ is a direct sum of unitary representations and the branching rules 
are obtained by normal holomorphic differentiation V k of degree k along the submanifold 
manifold G\jK\ in G/K. See also |fl4| for an abstract study. The representations appear- 
ing in the restriction will be also holomorphic representations and we are interested in the 
scalar ones (U„i, Gi). The branching of (U v , G) under H involves more analytic issues, 
and has been studied extensively |[T9l l23l l27l |28l [29]|. In particular it is proved that the 
complementary series (71^, H) appears in (U u , G) for certain pairs (/1, v). In the compact- 
picture of complementary series as distributional spaces on the boundary S = H/MAN 
of the unit ball H/L there is an intertwining operator 

J u :(<k^H)^(U u ,G) 

obtained by pairing the distributions on S in the complementary series with the reproduc- 
ing kernel of the holomorphic representation as a function on S x G/K, resulting in a 
kind of holomorphic extension of complementary series (71^, H) in ([/„, G). The same is 
true also for (7iy, H\) and (U v >, G\). The problem reduces thus to proving the operator 
(Jl)*V k J u in the diagram 

{U V ,G) -^-+ (U U/ ,G 1 ) 
{Ju'Y 

(tt^H) (V'^O 

is non-zero for appropriate /j, and /x'. This will be done by computing the operator acting 
on certain vector in (71^, H); see Theorem 5.1 below. 

It is clear that, compared with the method in [|26l where we study boundedness prop- 
erty of the restriction by K -type expansion, there are some advantages with the present 
approach as all the intertwining operators involved have been studied before and there 
also is flexibility of choosing the degree of differentiation. 
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The above procedure of holomorphic extension and differentiation can be applied to 
general spherical (not necessarily unitary) representations and we can find formally in- 
tertwining operators from spherical principal series of H = S0 (n,l;F) to that of 
Hi = S0 o (n — 1, 1; F). In the above compact realization of the representations as distri- 
bution spaces on the sphere S for H and respectively on the subsphere S b for Hi, those 
operators will be polynomials of normal differentiations orthogonal to the totally geodesic 
submanifold S b C S and of horizontal differentiations parallel to S b C S. The exact for- 
mulas can be rather involved. For the real case, H = SO Q (n, 1), Hi = S0 (n — 1,1), 
those operators have been found by Juhl [fi~2l using computations involving enveloping 
algebras and Verma modules. See also Kobayashi lfi~5l . In the last section we shall give 
for SOo(n, 1) a direct proof of our result using these differential intertwining operators 
in the non-compact realizations. 

The results in [1221 suggest that similar results should be true for the nonspherical com- 
plementary series representations of SO D (n, 1) and more generally for complementary 
series of all connected semi simple groups of real rank 1. 

The branching of complementary series is of interests in automorphic representations 

1221, go, a, a id. 

After a preliminary version of this paper was completed we were informed of the 
preprint [17] where they obtained a complete irreducible decomposition of (7r M , SO(n, 1)) 
under the subgroup SO(n — 1,1). The case n = 3 has been obtained earlier in mathe- 
matical physics litteratures; see e.g. [18J. 

Part this work was started during an AIM work "Branching problems in unitary rep- 
resentations", MPIM, Bonn, July 201 1. We thank AIM and MPIM for providing a very 
stimulating environment. We are also grateful to Roger Howe and T. N. Venkataramana 
for several conversations, and Bent 0rsted for drawing our attention to the work [12] . 

2. Spherical representations of connected semi simple Lie groups of 

real rank 1 

We recall very briefly some known results and fix some notation. Let F = R, C, EI be 

the real, complex and quaternionic numbers of real dimension i = 1,2,4. Let ¥ p ' q : = 

¥ p+q be equipped with the quadratic form |a;i| 2 + ■ ■ ■ + |x p | 2 — |a;p + i| 2 — ■ ■ • |x p+g | 2 for 

signature (op, uq). Denote SO (p, q; F) the connected component of group of F-linear 

transformations on ¥ p ' q preserving the form and having determinant 1, with F acting on 

the right. In the usual notation it is equal to SO (p, q), SU(p, q), Sp(p, q) respectively. 

Elements in SOq(p, q; F) and its Lie algebra so(p, q; F) will be written as (p + q) x (p + q) 

block F-matrices 

Oi a 2 

where a±, a 2 , a 3 , a 4 are of size pxp,pxq,qxp,qxq respectively. We shall consider 
subgroups SOo(r, s; F) in SOo(p, q; F), r < p, s < q, and fix the realizations as follows. 
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Write ¥ M = ¥ r © ¥ p ~ r © ¥ p © ¥"~ s . The group SO (r, s; F) consists of elements in 
SOo(p, q; ¥) fixing all vectors F p_r © ¥ g ~ s in the above decomposition. The Lie algebra 
so(r, s; ¥) consists of all elements in so(p, q; ¥) with the entries Oj being diagonal matrix 



°j 



bj 




with the obvious sizes of the blocks. The Riemannian symmetric space of SOo(p, q; F) 
will be realized as the matrix ball in M p>q (¥), 

SO (p,q;¥)/S(SO(p;¥) x SO(q;¥)) = {z G M m (¥);zz* < I}. 

The above identification defines a totally geodesic submanifold 

SO (r,s;¥)/S(SO{r;¥) x SO(s;¥)) c 5O (p,g;F)/ 1 S( 1 SO(p;F) x SO(g;F)). 

We fix now /J = SO (n, 1; F). The maximal compact subgroup L is equal to SO(n), 
S(U(n) x {7(1)), Sp{n) x Sp(l) respectively and and H/L is a Riemannian symmetric 
space of rank one. 

Let t) = [+q be the corresponding Cartan decomposition of the Lie algebra h. Elements 
in q are of the form 

"0 v 
v* 

with v G F n . Here v* = v T the conjugate transpose where a is the conjugation in F. We 
shall thus identify q with F". We fix 

H = ei eq = ¥ n ,eJ = [l,---,0] 

for F = R or C; for F = HI we let H be as in E6ll so that Ad(i7o) has eigenvalues 
±2, ±1, 0. Then a := Ri7 is a maximal abelian subspace of q. The root space decom- 
position of f) under H is 

f) = n_ 2 + n_i + (a + m) + ni + n 2 

with ±2, ±1, 0, if F = C, H, and with the convention that n 2 = if F = R. Here m C lis 
the zero root space. We denote by n = x\i © n 2 the sum of the positive root spaces. Then 
m + a + n is a maximal parabolic subalgebra of h. 

Denote M, A, N the corresponding subgroups with Lie algebras m, a, n. Then M = 
SO(n — 1), SU(n—l), respectively Sp(n— 1) x Sp(l) and MAN is a maximal parabolic 
subgroup of H . We normalize the /^-invariant norm on p so that H is a unit vector. The 
space S = K/M is then the unit sphere in p with M the isotropic subgroup of H G S. 

Let p = pn be the half sum of positive roots. Then 



p(Ho) 



n-l 

2 ' 


F = R 


n, 


F = C 


2n + l, 


F = H 
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and we shall identify p = p(H ). We let Hi C H be the the subgroup SOo(n — 1, 1, F) 
with the above convention of the notation. Let hi = li+qi be the Cartan decomposition of 
hi obtained under the Cartan involution on h restricted to hi. The subspace a = RH C q 
is also in qi, and we have the corresponding root space decomposition 

hi = tv-2,1 + ti-1,1 + (a + m n h) + ni,i + n 2 ,i, 

with 

n ±2 ,i = n ±2 , n ±1)1 = F"" 2 c n ±1 = F n ~\ 

and ni/ni,! = F. 

The corresponding subgroup Pi = P fl H\ = [L\ fl M)AN\ is a maximal parabolic 
subgroup of Hi, where iVi is the nilpotent subgroup with Lie algebra ni = ni ; i + n 2 ,i. 
The homogeneous space L/K fl M is a subsphere S\ := 5" t ( n ~ 1 )- 1 in the sphere S := 
K/M = S m ~\ 

(2.1) Si = L/K nil C S = K/M. 

Here i— 1,2 or 4. 

For n G C let 7r M be the spherical principal series representation of H induced from 
MAN consisting of measurable functions f on H such that 

(2.2) f(gme tHo n) = e~^f{g), me tH °n G MAN 

and /| G L 2 (L). In particular elements /| are invariant under the right action of M 
and thus f\ G L 2 (L/M) = L 2 (S). The group action of 7i>(7i) of i/ on L 2 (S) is 



a 6 
c d 



(2.3) Tr„(h)f(u) = \cu + rf|~ M /((an + 6)(cu + d)- 1 ), h~ l 
for / G L 2 (S). This can be derived for example by using that if 

h= a b , eH jU = k-H = kMAN G S = L/M. 

then 

(2.4) e -zpA{hk) = Jh ^ = | cn + d |-2p ? 

as proved in fTTTT 141 . Here e H ^ h ^ is the A-component in the Iwasawa decomposition g = 
ke A ^n of H = LAN of H, J g (u) is the Jacobian of g : u — > gu on S. 

The Lie algebra action (tt m , h) on the space L 2 (S)l of L-finite elements in L 2 (,S) de- 
fines a Harish-Chandra module which we denote also denote 7r M . 

If pi is real and < // < p(#o) provided that F = E, C and 2 < fj, < p(H ) for F = 
EI the representations (L 2 (S)l, tt^, h) are unitarizable and called complementary series. 
The unitarization of (L 2 (S)l, 7^, h) can be done through the Knapp-Stein intertwining 
operator. Its exact formula will not be used in our paper, but we recall it briefly as it 
motivates our definition of the J-operator in Theorem 4.2. 
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Let (x, u) = X\U\ + • • • + x n u n be the F-valued inner product on F n . Let 

(2.5) krf{x)=c „j s _m_ dn . 

We note that the integral is absolutely convergent for p < p for any bounded function / 
on S. As an operator on L-finite elements in L 2 (S) it has meromorphic continuation in p 
to the complex plane. Then 

(2.6) A 2p _ M : (L 2 (S) L , tv) -)■ (L 2 (S) L , n 2p -„) 

is a h -intertwining operator. A h-invariant Hermitian form on L 2 (S)l is given by 

(f,9)n= ( A 2 P -/J,#)- 

The decomposition of L 2 (S) under L is multiplicity free 0T), L 2 (S) = J2 T W T . In 
particular the operator A 2p _^ is diagonal under the decomposition, 

A 2p - M J = A„(r)/, for/eW T , 
with A M (r) explicitly found in [fTTT] . 

Theorem 2.1. [HI // 

(1) F = R, < fi < n - 1 = 2p H ; 

(2) F = C, < fi < In = 2p H ; 

(3) F = H, 2 < p < An = 1p H - % 

there is a positive definite ^-invariant form on L 2 (S)l defined on the isotopic component 
W T by 

\H\l = X,(r)\\w\\ 2 . 

Its completion (C M , 7r M , H) defines an unitary irreducible complementary series represen- 
tations of H. 

The unitary representations ix^ and 7r 2p _ M are unitarily equivalent. To avoid possible 
confusions we always assume that p < pjj, i.e. p is in the left half of the above interval 
for complementary series. 

3. Analytic continuation of holomorphic discrete series of unitary 
groups and their restrictions to a unitary subgroup 

We fix pairs (G, G\) as follows 

(1) F = R, (G, C7i) = (SU(n, 1), SU{n - 1)); 

(2) F = C, (C7, G x ) = (SU(n, 1) x SU(n, 1), SU(n - 1, 1) x SU(n - 1, 1)); 

(3) F = H, (G, d) = (SU(2n, 2), SU(2(n - 1), 2)). 

In this action we discuss the branching of analytic continuation of the holomorphic dis- 
crete series of G under under restriction to C7i . Results for F = C, i.e., the tensor product 
case, are trivial generalization of those for R and will be treated only very briefly. 
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The group G is a Hermitian Lie group and its symmetric space G/K will be realized 
as a bounded symmetric domain D. 

More precisely we view the complex matrices z in 

M 2n , 2 (C) = M 2 , 2 (C) © ■ • • © M 2 , 2 (C) 

as column vectors z — [z±, ■ ■ ■ , z n ] T with each Zj being 2 x 2-matrices and define 

'C n CaseM 

^C"©C^ CaseC 
M 2nj2 (C) CaseH. 



The group G acts on 

by 



D = {zeV;zz* </} 



</~ 



(az + 6)(cz + <i) , g 



a b 
c d 



GC7, 



in the cases of M. and H, it acts on the second component (z±, z 2 ) G -D as anti-holomorphic 
mappings in the case C. 

The holomorphic discrete series of G are realized as weighted Bergman spaces of holo- 
morphic functions on D and are determined by their reproducing kernels. Let 

{l-(z,w) Case R 

(1 - {z x ,wi))(l - {w 2 ,z 2 )) CaseC 
det(l -w*z) Case H. 

Note that the /i-function for F = R can also be written as det(l — w*z). 
When v is in the set 

(n, oo) Case M, C 

(2n + l,oo) CaseH 

the kernel h(z, w)~ u is the reproducing kernel of a weighed Bergman space T-L u with a 
probability measure on D, i.e. the define discrete series representations of the universal 
covering of G. 

The group G acts on T-L u via 



U v (g)f(z) = det(cz + d)^f{g- l z), g~ x 



a b 
c d 



defining a (projective) unitary representation (U V ,G); in the case of C it is the tensor 
product. The set of all v > for which h(z, w)~ u is positive definite (also called Wallach 
set) is well-known for general bounded symmetric domains [0 [6l |20l [25j ; it consists 
of an open interval and discrete points and we shall only need open interval. In our 
parametrization we have 
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Lemma 3.1. The functions h(z, w)~ u is positive definite ifv is in the set 

(0, oo) Case E, C, 
[1, oo) Case HI. 

and it determines a unitary (projective) representation (U u , G) on a Hilbert space of 
holomorphic functions on D with reproducing kernel h(z, w)~ u . In particular the repre- 
sentation (U U ,G) is unitary and not in the discrete series ifv is in the set 

(0, n) Case R, C, 

[l,2n+l) Case H 

We denote the analytic continuation also by (H u , U u , G) = {H V (D), U v , G). In this 
paper we are concerned with representations (T-L u , U u , G) whose parameter v is in a subin- 
terval of the above interval, i.e.,it is not a discrete holomorphic representations. 

For the case C with G = SU(n, 1) x SU(n, 1) we let H U (D ® D) = U V {D) ®U V (D) 
and (H U (D ® D), U v , G) be the tensor product of (U u ® U~ v , SU(n, 1) x SU(n, 1)), for 
i/>0. 

To find the branching of (H^, U u , G) under Gi we realize D\ = G\jK x as a sub- 
manifold of D = G/K as in §2. For any polynomial p we denote d p the corresponding 
differential operator. Let 

{z n , CaseM 

z n w n , CaseC 
det(z n ), CaseH. 
The polynomial Q on V, as tangent space of D — G/K, is invariant under the subgroup 
K\ C K. 

Theorem 3.2. Suppose that v satisfies the assumptions of 3.1. 

(1) The restriction ofof{U v , G) under G\ is a discrete sum of holomorphic represen- 
tations ofG\. 

(2) The holomorphic representation (U u+ j, G\) for a nonnegative integer], is a dis- 
crete summand of representation (U u , G), restricted to G\ 

(3) A non-zero G\-intertwining operator from {U v , G) to (£/„+•/, Gi) is given by 

W:f^d Q] f\ Dl . 

Proof. Since ([/„, G) is a highest weight module the first assertion follows from [fl4l . 
Following the ideas in [0 to prove the second and third assertion we realize in the case R 
the group Gi C G acts on % v via 

a b 



U l/ (g)f(z',z n ) = (cz'+d)^f((az'+b)(cz'+dy\z n (cz'+dy 1 ), g~ l = ~ ^ G G 1 . 

Thus (d^) Zn=0 is an intertwining operator 

di(U u (g)f)(z',0) = (cz' + d)-^ (dlf) ((az' + b)(cz' + d)-\0) = n u+j (g) (#/(•,())) . 
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The case C follows trivially. In the case HI the intertwining property is a simple conse- 
quence of 

det(d n y(f(Az n )) = det(Ay(det(d n yf)(Az n ) 
for any function / on M 2i 2(C). □ 

4. The branching of (U„, G) under H. The appearance of the 

COMPLEMENTARY SERIES (7^, H) 

We consider H C G, Hi C G\, written as (H,Hi) C (G,Gi), where 
(H, H t ) = (SO (n, 1; F), SO (n - 1, 1; F)) 

• If F = K. then (if, Hi) C (C7, Gi) is the fix point set of the unitary groups under 
complex conjugation. 

• If F = C the subgroup pair (H, H{) C (G, (?i) is the diagonally embedded. 

• If F = EI we first identify the quaternions HI as the real form 



U= { 



a + ib c + id 
— c + id a — ib 



C M 2i2 (C) 



in M 2i 2(C), i.e. F = EI is the set of elements in M 2j 2(C) fixed by the real involu- 
tion J : z = x + iy — > x — iy in M 2j 2(C). Observe that under this identification 

dete= |e| 2 . 

The involution defines a corresponding J on M 2+2n2+2n (C). The group H = 
Sp(n, 1) is then the realized as the set of elements in SU(2n, 2) C M 2+2 „ i2+2t i(C) 
fixed by J. 

In this notation the complex-valued polynomial det : M 22 (C) — ► C is the unique 
Sp(l) x Sp(l) -invariant holomorphic polynomial of degree 2. The complex matrices z in 
-^2n,2(C) are then complexified quaternions column vectors z — [z±, ■ ■ ■ , z n ] T . Consider 
the vectors z — [zi, ■ ■ • , z n ] T with quatemionic entries, i.e., in the real space HP. Then 

p(z) = det z\ -\ — • + det z n = \zi\ 2 + ■ ■ ■ + \z n \ 2 = \\z\\ 2 , 

is invariant under Sp(n) x Sp(l) with Sp(n) acting on the left and Sp(l) on the right. 
Thus the holomorphic polynomial p(z) = det Z\ + ■ ■ ■ + det z n is also invariant on the 
complexification M 22 (C) since H" is a totally real subspace in M 22 (C). 

The main result in the section is 

Theorem 4.1. Every complementary series (n^, H), fi < pu, of H is a direct summand 
of a holomorphic representation of G. 
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To obtain the precise formulas for the branching we now recall the branching of (T-L u , U u , G) 
under H. It has been studied in ll23l l27ll29l ; see also |fT9l for some general higher rank 
cases. Let X = H/L be the symmetric space of H as in §2. The identification of H 
with a subgroup in G induces an imbedding of X in D = G/K as a totally real totally 
geodesic submanifold. 

Let R:U V ^ C°°(X) be the restriction 

OT(x) = i F ^ X £ X 
[0 otherwise 

Define the weighted restriction R v : % v — V C°°(X) by 

R u F(x) = (1 - (x,x)) x RF(x) = (1 - (x,x)) x F(x),xe X C D, 

where 

ff, ¥ = R 

A = I v, F = C 

[ v, F = H. 

The functions in (%„, G) are holomorphic and thus smooth so R and R v are well-defined. 
-Ry is an if -intertwining operator 

R l/ :(H l/1 G)-^(C 00 (X) 1 H) 

with the regular action of H on C°°(X). 

We denote by H V (D) the space of holomorphic functions on some neighborhoods of 
D; the group G acts % V {D) via the action [/„. In particular elements in 1-L U (D) have 
smooth restrictions to S, where S is the sphere in (12.11) . i.e., the boundary of X. By abuse 
of notation we denote this restriction also by R. We observe that for 

ju, if F = M 
^ ~ 1 2v, if F = C, H, 

R is an iJ-intertrwining operator 

(4.1) R : (H V (D), U„, G)\ H -> (C 00 ^), tt m , if), 

If F = R this follows immediately from the definition. For the case F = C we have 

R(U u (h,h)f)(s) = (U l/ (h,h)f)(s) 



= (cs + d)~ v (cs + d)- u f((as + b)(cs + d)" 1 , (as + b)(cs + d)- 1 ) 
= \cs + d\- 2u Rf{(as + b){cs + d)" 1 ), 

where /i _1 is as in (12.31) . The intertwining relation follows from (I2.3I ). The case of F = EI 
is a similar argument using the observation that det(g) = \q\ 2 for any quaternionic number 
q, with det(g) being the determinant of the corresponding 2 x 2-complex matrix q. 
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Let R* be the adjoint of R v considered as a densely defined operator on L 2 (X) = 
L 2 (H/L) with the G-invariant measure di(x). Then R V R* V is an integral (generally un- 
bounded) operator on L 2 (X) 

(4.2) R v R*J{x) = [ B u (x,y)f(y)dL(x), B(x,y) = ( (1 ~ ( *' g))(1 ~ { ^ x) ) 

Jx V 10-- {x,y))r 

This operator is called Berezin transform [|27], and the representations defined by the 
kernel is called the canonical representation [|2~3l . The spectral decomposition of R V R* V 
is given in [|23l . As a consequence we get the branching of (U„, G) under H. There are 
finitely many discrete components and we shall need the first one (compared in terms of 
their eigenvalues of the Casimir operator) of them. 

Theorem 4.2. Consider the holomorphic representaton (H U (D), U u , G). Let 

u, ue(p,pH) = (P,^), F = M 

{2v, l / 6 (0,^) = (0,f), F = C 

2i/, l / 6 (l,^) = (l,^), F = e. 

Then the complementary series (jr^H) is a discrete summand of the restriction of in 
(Ji v (D), U v , G) to H. Moreover it has multiplicity one in the restriction. In particular 
the operator 

R : n„(D) -)• C°°(S) 
extends a H -invariant bounded operator from T-L u onto n^. 

Note that the relation between the parameters v and \x is determined by the intertwining 
property (4.1). 

We shall need concrete constructions of the intertwining operators. The following 
result can be derived from [|29] Theorem 5.2], where general higher cases are treated. We 
present the construction and include a proof of the theorem here in the rank one case. 

Theorem 4.3. Let v be as in Theorem 4.1. Identifying the dual space ofir^ with 7T2 P - fl 
via the standard G-invariant paring in L 2 (S), 



(f,9) = / f(x)g(x)dx 
>s 



the dual J u = R* of R : % v — > 7r„ is given by 



Ju ■■ (K2 P -»,H) -+ (U V ,G)\ H , J v f(z) = f TT^ds, z e D' 

The operator J u is defined as an integral operator with kernel h(z,s) v , the reproducing 
kernel of H U (D). We need its expansion [6] into A"-types of of H U (D); for the cases of 
F = M, C it is just the binomial expansion. 
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Let V(V) be the space of polynomials on V . It has [JVl [6J an irreducible decomposition 
under K, known as Hua-Schmid decomposition 

fE^=o^(C n ) CaseR 

V{V) = I E" lim2=0 ^(C") ® P m (C») CaseC 

l k Z^m=(mi,m 2 ),mi>m2>0 ' 12. Case H 

where V m (C n ) is the space of polynomials of degree m, and for the case EI the space Vm 
is an irreducible representation of K generated by z^ 1_m2 det(zi) m2 for z = [z±,--- , z n ] T e 
V with 2?i G M 2! 2(C) (of highest weight —miji — m 2 72 with 71,72 being the Harish- 
Chandra roots). 

Denote (a) m = YlT=o ( a + J) an d («)m = TlT=o ( a + i)( a — 1 + j) me Pochammer 
symbol. Let K m (z, w) and K^z, w) be the reproducing kernel of the irreducible spaces 
V m and Vra with the Fock space norm. We have then 

K m {z,w) = —Az,w) m 

ml 



in Case R and 



g-(«.«)= ,yf tr -— ( ^ det(^)- 

— (2ra)m m 1 -m 2 + l 



in Case H, the first formula is elementary whereas the second can be obtained from 11291 
Lemma 3.1]. The Faraut-Koranyi expansion is then the binomial series 

00 ,. 

h{z,w)~ u = )(u) m K m (z,w), K m (z,w) = —(z,w) m 
*■ — ' ml 

m=0 

for F = R, and 

00 

h(z,w)~ u = y](y)m^aa(^w), 

m>0 

if F = EL Equivalently this can be rephrased as 

(4-3) ll/lk=Mm||/||2. 

The case F = C is deduced from the real case. 
We can now sketch the proof of the theorem. 

Proof. The formula for J v is obtained using the reproducing kernel property. Since 
(7T2p-/i, H) is unitary and irreducible we need only to check that J u maps the constant 
function 1 in (7^-^, H) into a non-zero element in (U„, G), namely we have to prove 
that 

(4.4) f(2):=J „ 1 = | s _L_, s , 2eD 

is in (U U ,G). 
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F = R. Using the power series expansion of h(z, s)~ v we find 



^ m\ J s 



Clearly the integral J s (z,s) m ds is zero unless m = 21 is even, in which case it is a 
constant multiple of the polynomial q(z) 1 := (zf + ■ ■ ■ + z^) 1 . The constant can be 
evaluated at z — e\, namely 

f(z, s) 2l ds = q{z) 1 ! sfds = C n B(l + 1 ^)q(z) 1 

= C n B(l + \, ^Hz) 1 = c n ^^ q {zy, 

where C n and C' n arc constants depending only on n. Thus 

nz) = C n ^ WWT ^q(z) 
and the membership of F in (U u , G) is determined by the convergence of square norm 

II ETII2 _ r <2 ST^, (y)2l £(j + 2) \2|| f ||2 _ r >2 ST^ A^hl r (^+ 2) \2 I II /||2 

11 \\u "Z^^( 2 /)!r(/ + f) J m " v n Y l (2/)ir(/ + f) J oV 



The square norm ||g ||jr is 



n . 



l? , ||^ = 2 !B «Qi = ^ 



r(|)(20! / n, 



■2" r(| + z) v 2- 

Simplifying we find 

l2 ^,v->w r d+§: 



LF 



n Y ( 2/ ) ! r^+t) tr/- 



which is convergent if and only v < ^^ = p. 

F = C. This case is studied in [O where a construction for all discrete components in 
the branching of (C/ M , G) \h is obtained. The function F above is 

^0 W™ m! 



and 

II Z7l|2 



V^A^)m( Z7 )m N 2 dim (^ 



m=0 (n)m W? m 

where A\m{V m ) ~ m™ _1 is the dimension of the polynomial space P m (C n ). Here we 
have used (14.31) applied to the tensor product. Each term is 

Ml .. _ , 1 „_, 1 



^dim(7> m ) ~ -j— -m 



n-l 



ri)1 ' ' m 2n ~ 2v m n+l~2u 



I III 



and the series is convergent if an only if 2v < n = p 0u (n,i)- 



14 B. SPEH AND G. ZHANG 

F = H. Similarly F(z) = ^ m (^)m Is ^mk z i s )ds, and the polynomial J s Km{z, s)ds 
is an L-invariant polynomial in ?„. It is proved in ||29l Lemma 3.3] that this space is zero 
unless m = (m, m), in which case it is one-dimensional, namely given by the polynomial 

q(z) m = (det z x + • • • + det z n ) m . 

[ K nk (z,s)ds = C nk q(z) m 
Js 

We evaluate it at z = [i 2 , • • • , 0] T . The left hand side is 



det( Sl ) m ds= / \s 1 \ Im ds = C \yr m (l-\y\T dv 

s Js Ju 

= C f r 2m (l - r 2 ) 2n - 3 r 3 drdy 
Jo 

= c „ (m+1)! 

(2n - l) m 

with a normalization constant C depending only only on n. The square norm of F in 
(^,G)is 



(m + 1)! 



2 



||p||2_ r - V^ (iA 2 l _V!±Z±1L\ \\n m \\ 2 

\W\i-c ^ ^[j^—^) hh 

m=(m,m) 

The Fock space norm ||g m ||^- of p m can be computed by elementary differentiation. We 
have the following (Caley-Capelli type) identity 

q (d)q m = m{m - 1 + 2n)q m ~\ 

from which it follows 

\\q m f r = m\{2n) m 
The square norm of 

oo _. 

II 77-11 2 V^ 



m 2n+2n-l-(2n+2v-l) 
m=l 



which is convergent precisely when 2v < 2n + 1 = p s p( n ,i)- D 

5. Discrete summands in the restriction of (7r„, if) to the subgroup H 1 
We prove now our main theorem of this paper. 

Theorem 5.1. 

A.) Assume that F = M and < p < pu- If p + 2? < p Hl j > 0, then (7i^ +2 j, Hi) is a 

subrepresentation of the restriction o/(7r M , H) to H\. 

B.) Assume that F = C and < p < pu- If p + 2j < p Hl j > 0, then (7r M+2 j, Hi) is a 

subrepresentation of the restriction o/(7r M , if) to iii. 

C.) Assume that F = EI and 2 < p < p H . If p + 2j < p Hl j > 0, then (ir^j, iii) w a 

subrepresentation of the restriction 0/(717,, if) to Hi. 
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Actually we will prove in the next section that for F = R, < /i < p#, /i + j < pn t 
j > 0, then (itn+j, Hi) is a subrepresentation of the restriction of (71^, H) to Hi. Our 
proof below uses explicit differentiations of d 3 > on (zf + ■ — h z 2 n _ x + z 2 ,)™ evaluated at 
z n = which gives trivial result if j' is odd. Presumably the same proof will also work 
for odd f by replacing the function 1 below by x\. 

Proof. Recall that \x is defined as a function of v by 

(u, i/6(0,^) = (0,2fi), F = R 

(5.1) n=l2v,ue(0,f) = (p t %), F = C 

(2u, l /e(l,^) = (2,^i), F = ft 

We consider the operator, recalling T>* in Theorem 3.2, 

(5.2) Tj = {J v+j ,)*V? J v = RVi'j u , 

where f = 2j for F = R and f = j for F = C, H. (We choose f = 2j for F = R 
to ensure that (Tjl, 1) below is non-zero.) Note that it should be clear from the formula 
the spaces on which the J-operators are defined. The operator Tj is bounded and Hi- 
intertwining 

Tj =: (C 2 p H -v,ir 2 p H - u ,H) A (H U ,U V ,G) -> (H u+ j>,U u+j >,Gi) -> (C^'),^^'),^) 

by Theorems 3.2 and 4.2, where /i(j) is obtained from v+f via the above correspondence 
/i -H- za Composed with the equivalence A 2pH _ !/ of (C u , ir u , H) and (C2 PH -u, ^2 PH -v, H) 
we get a bounded Hi -intertwining operator 

TjA u : (C u ,n u ,H) -»■ (Cfj,(j),7r^j),Hi) 

which preserves the Hilbert space norms on H and Hi. Thus we need only to check that 
Tjl j^ 0. We compute the pairing (Tjl, 1) in Theorem 4.2. 

(T j l,l) = (Vi'j 1/ l,J2 PHl ^( j )l)H u+j ,. 

The functions J v 1 and J2 PH _ M (j) 1 are found in (14.41 ) in the proof of Theorem 4.2 (for the 
bigger group pair (G, H)). We shall find V k J v l following the computations there. 
F = R. In this case V 2j J v \ is, apart from a positive normalization constant, 

r{l+1 *]mi(l) q (z>,o^ 

l>3 x ' 2 ' 



(^rxz+j) m 
^(2/)!r(/ + f) (2j H2jr ( "' U) 



for 



whereas J 2pHi _ At(i )l is 



(^)(^0) = (2 J )!( 2 / J )^ , ,0)^ 



py^ _ r r (2p gl -Mi))2 fc r(fc + |) . 
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The inner product (V 2 iJ v l, J^l)u 2p _ U) , is then, up to a positive constant, 



f* Wh m + j) ( i \ (2 PHl - mu r(k + 1; 

2- t (2l)\T{l + ^ 3) \2j) (2k)\ r(* + §; 



(q(z') k ,q(z') k ) 



l-j=k=0 v ~~ / ~ " v ~ ' 2 > V " J/ v "" v ' ■*" v " ' 2; 

which is a (convergent) sum of positive terms and thus is non-zero. 

F = C, H. The computations are similar. We need only to note that for the H-case, 



^(z^O) = (™)(j + l)\jlq(z',0r-i 



J 
with the coefficients being also positive. □ 

We note that our approach above can also be used to treat the complementary series 
for H = -F 4 (_ 20 ) ifTOl under the symmetric subgroup Hi = Spin(8, 1) by considering the 
Hermitian Lie groups G D Gi with G = #6(-25)> G\ = Spin(8,2). However no more 
extra discrete component will be discovered except the one obtained using the restriction 
method in ll26l . namely the zero degree of differentiation in the above method. This is 
due to the rather short interval of complementary series of H = -F 4 („ 2 o) • More precisely 
the eligible range of the parameter v of holomorphic representations (U u , G) for which 
discrete components under H do appear is the interval (3, y). Here the lower bound 
3 corresponds to the first discrete Wallach reducible point for G, and the upper bound 
corresponds to ^- which is the critical point where the spectral symbol of the Berezin 
transform RR* in (14.21) has a pole ll26l (6.1)]. The same range of v for ([/„, Gi) for the 
pair (Gi, Hi) = (Spin(8, 2), Spin(8, 1)) is (3, |). (The pn and p Hl are 11 and 7 with 
the same normalization of split Cartan in ^ C f), the half values y and | in the intervals 
are due to the double rank of G and G x of that of H and Hi.) The symmetric domain 
Di = Gi/Ki is in Vi = O c as a subdomain of D = G/K in O^ = V\ © V 2 . A normal 
differentiation by any Ki invariant polynomial on V2 on the subdomain Di = GijK\ 
will result in intertwining operator from (U v , G) into (U u >, Gi) with v' > v + 1. But for 
v > 3 we have 1/ > 4 > | which is outside the eligible range (3, |). Presumably this is 
the only possible discrete complement. 

6. Explicit realization of discrete components via intertwining 

differential operators 

For the real case a direct proof of the appearance of the discrete components in the 
branching can be obtained using the explicit formula for intertwining differential opera- 
tors [12] for spherical representations of H and Hi. The operators have an easier expres- 
sion in the non-compact realization. In this realization our proof is a simple application of 
the Cauchy-Schwarz inequality, and it also yields an explicit construction of the discrete 
components. 

The complementary series (C M , 71^, H), H = SO (n, 1), can be realized as distribution 
space on n = M™" 1 ; see e.g. |]33|. Initially it acts on a subspace qf (M 71 " 1 ) of C^M™" 1 ) 
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obtained from the restriction to IR n ~ 1 = N~ of smooth functions on G; alternatively it 
can be obtained by a a Cayley transfrom from S to M n_1 and the if-action on the space 
C°°(S). The action of P" = MAN' on the space qf (M"" 1 ) is 

Kp(m)f(x) = /(m^x), 7r M (a)/(x) = a'^fia^x), fCp(y)f(x) = f(x - y), 

for 

m G SO (n - 1), aeR + = A, ye W 1 ^ 1 = N~. 

The Weyl group element w acts as an inversion 

It 1 

7T»/(*) = /(- — )( — )" 

\x\ l \x\ z 
The Knapp-Stein intertwining operator is 

/-► / f(y)\x-y\- 2 ^ l ^dy 

as a meromorphic continuation of integral operators. Let A n _i = df + ■ • ■ + d^_ x be the 
Laplace operator on M n_1 and T the Fourier transform. Then (C M , 7r M , H) is the comple- 
tion of qf^M™- 1 ) with the norm 

(6.i) ii/ii* = w^^^m, = I \jFf{i) ner- 1 -^ 

for < /i < !1 ^- = p. See e.g. []24l Theorem 2.1], our parameter /i being their n — 1 — A. 



The explicit form of 7r M is well-known and will not be used here. Correspondingly the 
representation (C M , n^, Hi) is realized on M n ~ 2 , considered as the subspace {(x, 0); x G 
R n - 2 }ofR n_1 . 

We recall the result of Juhl [fT2l Ch. V] on intertwining differential operators, with our 
parametrization of the spherical representations. 

Theorem 6.1. Let n > 2 and /i G C. T/zere ex/sto an non-zero intertwining H\ operator 

V^ : (C^CR"- 1 ),^,^) -► {C^iW-^^^H,) 
of the form 

(6.2) ^/M) = l^C^dizfA^j) (y,0) 

\2k<j J 

where C Mj & are constants depending on /i, m normalized by C M] o = 1. 

We give now an independent proof of Theorem 5.1 for the real case. 

Theorem 6.2. Letn > 2, < /i < pu = ^^ and p + j < pu x = ^-jp- Then the operator 
D^j is a non-zero Hi-invariant bounded operator from (C M , H) onto (C M+J , Hi). Thus the 
complemetnary series (C^+j , H i ) appears as a discrete component in the decompostion 
of (Cfj,, H) under Hi. 
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Proof. Note that the second claim follows from the first one by a standard argument. 
Indeed the adjoint V* ■ of V^j is then a non-zero bounded i^-invariant operator from 
(C^+j, Hi) to (C M , H). Its polar decomposition defines an Hi -invariant partial isometry. 
But since (C^+j, Hi) is irreducible it is an isometry. 

We denote P the Fourier transform on W 1 ' 2 . Let / G C^M"" 1 ) and g := D^f. We 
shall prove that 

\\g\\l +3 ■= [ \vr 2 - 2{fl+J Vg(v)\ 2 dv < cy/iij. 

By the Fourier inversion formula we have, writing the dual variable £ G IR™" 1 as £ = 

/(y, x„_i) = C n f Ff( v , i n -i)e^^ x ^^d v di n _i, 

where C n is the normalization constant for the Fourier inversion. Performing the differ- 
entiation V^j evaluated at (y, 0) we get 

g(y) = V,J(y, 0) = C n [ Ff( V , Cn-i)e iM E( V , ^O^n-i 

= C n f e 1 ^ ( f Tf(r), Cn-i)E(r,, £„_i)d£„-i J ^ 
where E(£) = E(r), £ n -i) is the Fourier multiplier corresponding to (I6.2K 

^(o = ^^n-o = E CM(-i^r) fc (-^-ir 2fc - 

2fc<j 

The Fourier transform T^g of g is 



rg{ri)=C / 7^/(77, £ ft _i)S(77,£ ft _i)de ft _i = C / .F/(0£(0dfn-i 

Jr. Jr 

where we have written £ = (77, £ n _i). We apply the Cauchy-Schwarz inequality, obtaining 

\^g(v)\ 2 < jf ^/(oner 1 -^^ jf i^)i a j|p^d^i- 

Now the polynomial i£(£) is a sum of terms of the form £^jj \r]\ 2k , < 2k < j, and we 
apply the triangle inequality to the second integration, 

The each term can be evaluated by a change of variables, £ n _! = |?7|ti, 



|£|n-l-2/j •'•"■- 1 ^\n-2-2fi-2j J^ H + M 2)n-1-2 M ~ |^|n-2-2/i-2j ' 

Here we have used the assumption that j < j so that n— 1— 2/x— 4fc > n — l — 2^—2j > 1 
and the integration 



u 4k 

C = I - —~^du < OO 



(1 + u 2 )™- 1 - 2 ^ 
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is convergent. That is 

f 1 1 

and 

lA^r < g „ 2 1 _ 2At _ 2j / i^/(o i 2 ier- x - 2 ^-i. 

I 7 ?! </r 

Integrating over 77 £ M n ~ 2 we find 

\\g\\l +j = [ \^g(v)\ 2 \vr 2 ~ 2 ^ <c f [ ^/(orier 1 -^ = cii/ii 2 

JR"- 2 JR™-2 JR 

completing the proof. □ 



For j = 0, i.e. V^o being the restriction, the above result is proved in [1221 by consid- 
ering the dual map of the restriction. Note the above proof is somewhat simpler. 

Note the standard L 2 (M n ~ 1 )-pairing / <g) g —} (f,g) is invariant under it^ ® itzp-^ and 
we may rephrase the above theorem as a realization of the discrete components. 

Corollary 6.3. Let \x and j be as above. The representation (n n _ 2 -^ + j), Hi) appears in 
the branching of (7r n _i_ M , H) under Hi and an explicit realization of the subrepresenta- 
tion is the distribution space 

{F;F(y,x n _i) = ^ C^cfcf A k n _ 2 (f{y)6{x n „i)), f E (rr n _ 2 _ {p+jh Hi)}, 

2k<j 

where 5 is the Dirac's delta function. 

We may also use the version of V^j in the compact picture and to prove Theorem 
6.2 directly using the method in ll26l . This involves rather detailed computations of nor- 
mal differentiation of spherical harmonics. The above proof using Fourier transform is 
apparently easier. 

Finally we study the limit case /1 = using the operator V^j. Assume n > 2 is an 
odd integer. The Harish-Chandra module of A'-finite elements in C°°(S) has the constant 
functions C as its submodule and the quotient space is unitarizable, with the unitary norm 
obtained from that of C M by continuation (or residue depending on the normalization); see 
ifTTTl . We denote the quotient space by Wq. Now the norm 



(6.3) 11/112 = 1^/11^= / i-F/corier^ 

is the continuation of (16.11) at // = and it vanishes on the constant function. Thus it is 
the unitary norm on Wo- See also |[24l Theorem 2.2]. Using the same proof as above we 
have the following result 

Theorem 6.4. Let n > 2. Then the operator V j is a bounded from Wo onto (Cj, Hi) 
for 1 < j < " [ ^ L . Thus (Cj,Hi), 1 < j < f± Y L , appear as a discrete component of 
(Wo,H) Hl . 
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